Toward a theory of information processing

Information processing theory endeavors to quantify how well signals encode information and how well systems, by acting on signals, process information. We use information-theoretic distance measures, the Kullback-Leibler distance in particular, to quantify how well signals represent information. The ratio of distances between a system’s output and input quantifies the system’s information processing properties. Using this approach, we derive the fundamental processing capabilities of simple system architectures without detailing the component systems and the signals they process.